Dynamic hierarchical reserved resource allocation

ABSTRACT

A method includes receiving a request to allocate an instantiation of a network function and information indicative of resource needs of the instantiation. The resource needs include at least one resiliency requirement. The method includes computing a resource map comprising a global tier and a regional tier and comparing the resource needs with the resource map to determine an allocation solution. The method also includes, based on the allocation solution, allocating resources to the instantiation. The resources include a first resource of the global tier and a second resource of the regional tier.

TECHNICAL FIELD

This disclosure relates generally to cloud network resource managementand, more specifically, to reservation of computing resources fordifferent levels of resiliency within a local cloud environment andbetween multiple geographically distributed cloud environments.

BACKGROUND

To provide a service or application (generally “an application”) usingvirtualized network platforms, a set of one or more virtual networkfunctions (VNFs) and physical network functions (PNFs) may beinstantiated on general purpose hardware by allocating computingresources to that application. These computing sources may be located inlocal datacenters, geographically redundant datacenters, or acombination thereof.

The problem is that there is not yet a solution for the orchestrationallocation and relocation of reserved computing resources that protectagainst the potential loss of different levels of local andgeographically distributed computing resources. In the case of a varietyof multiple failures of the normal computing capacity in localdatacenters or geographically redundant data centers, there is not yet adefinition of a hierarchical and rule-based algorithm to prioritize therelocation of the reserved computing resources, according to rulesestablished by a cloud infrastructure operator for the service.

This disclosure is directed to advancing the state of the technologicalarts by solving one or more of the problems in the existing technology.

SUMMARY

In an aspect, a cloud orchestrator may include a network connection forconnecting to a cloud network, a processor communicatively coupled tothe network connection, and memory storing instructions that cause theprocessor to effectuate operations. The operations may include receivinga request to allocate an instantiation of a network function andinformation indicative of resource needs of the instantiation. Theresource needs may include at least one resiliency requirement. Theoperations may also include computing a resource map of the cloudnetwork. The resource map may include a global tier and a regional tier.The operations may include comparing the resource needs with theresource map to determine an allocation solution and, based on theallocation solution, allocating resources to the instantiation. Theresources comprise a first resource of the global tier and a secondresource of the regional tier.

In another aspect, a method may include receiving a request to allocatean instantiation of a network function and information indicative ofresource needs of the instantiation. The resource needs may include atleast one resiliency requirement. The method may include computing aresource map comprising a global tier and a regional tier and comparingthe resource needs with the resource map to determine an allocationsolution. The method may include, based on the allocation solution,allocating resources to the instantiation. The resources may include afirst resource of the global tier and a second resource of the regionaltier.

According to yet another aspect, non-transitory computer readablestorage medium storing instructions that cause a processor executing theinstructions to effectuate operations. The operations may includereceiving a request to allocate an instantiation of a network functionand determining resource needs of the instantiation based on a userinput. The resource needs may include at least one resiliencyrequirement. The operations may include computing a resource mapcomprising hierarchy of datacenters and comparing the resource needswith the resource map to determine an allocation solution. Theoperations may include based on the allocation solution, allocatingresources of the datacenters to the instantiation. The resources maysatisfy the at least one resiliency requirement.

BRIEF DESCRIPTION OF THE DRAWINGS

In the following description, for purposes of explanation, numerousspecific details are set forth in order to provide an understanding ofthe variations in implementing the disclosed technology. However, theinstant disclosure may take many different forms and should not beconstrued as limited to the examples set forth herein. Where practical,like numbers refer to like elements throughout.

FIG. 1A is a representation of an exemplary network.

FIG. 1B is a representation of an exemplary datacenter.

FIG. 1C is a schematic of the relationships of an orchestrator used toallocate and relocate instantiations within a network.

FIG. 2 is a representation of an exemplary method that may be used toallocate resource to instantiate a network function.

FIG. 3 is a schematic of an exemplary device that may be a component ofthe system of FIG. 1C.

FIG. 4 depicts an exemplary communication system that provide wirelesstelecommunication services over wireless communication networks withinwhich a network function may be instantiated using the disclosed systemsor methods.

FIG. 5 depicts an exemplary communication system that provide wirelesstelecommunication services within which network functions may beallocated using the disclosed systems or methods.

FIG. 6 is a diagram of an exemplary telecommunications system in whichthe disclosed systems or methods may be implemented.

FIG. 7 is an example system diagram of a radio access network and a corenetwork within which network functions may be allocated using thedisclosed systems or methods.

DETAILED DESCRIPTION

Within a datacenter, there are multiple levels of computing capacity,such as individual computing hosts, and availability zones, which mayfail or become unavailable for maintenance reasons. Failure may arisefrom location-specific issues like power outages. Additionalgeographically diverse datacenters provide another level of availabilityin the case of the loss or reservation of other datacenters. In the caseof multiple failures in local data centers and geographically diversedata centers, a unified resource allocation and relocation algorithm,designed based upon rules defined by operators, can be used to handlethe availability of a service.

This disclosure is directed to a cloud orchestrator that receives rulesand implementation requirements, compares those requirements with theavailable resources across multiple datacenters, and allocates resourcesof those datacenters to a VNF based on the requirements. Further, in theevent of failure or unavailability of resources, the cloud orchestratorrelocates the VNFs to different resources.

FIG. 1A is a representation of an exemplary network 100. Network 100 maybe or include a software defined network (“SDN”) in which elements ofnetwork 100 are distributed across multiple datacenters 102.

Within data center 102, there may be multiple levels of computingcapacity, such as individual computing hosts and availability zones.Further, certain portions of computing resources within datacenter 102may be reserved, such as for specific services. The functionality andconfiguration of a datacenter 102 is discussed in more detail below withreference to FIG. 1B.

Within the collection of datacenters 102, there may be hierarchicalcontrol. This control may be present in all interactions of suchdatacenters 102, or it may exist based on the specific needs of aninstantiation of a network function. This hierarchy may be representedin multiple tiers. The higher the tier, the higher the latency forapplications, particularly with respect to their interactions with edgedevices and end users. Thus, low-latency applications may beinstantiated on lower tiers in order to meet latency requirements thatotherwise could not be met if instantiated on the higher tierdatacenters 102.

At the highest point of the hierarchy, a global tier 104 may providecentralized control. Datacenters 102 in global tier 104 may communicatewith (and exert some control over) datacenters 102 in the next lowertier, the regional tier 106. The regional tier 106 may be desired forgeographic distribution of datacenters 102. In turn, datacenters 102 inregional tier 106 may communicate with (and exert some control over)datacenters 102 in the next lower tier, the edge tier 108. Edge tier 108may control and manage a network edge 110, through which end devices 112connect to and interact with network 100.

FIG. 1B illustrates an exemplary configuration of datacenter 102. Eachdatacenter 102 may comprise one or more racks 114. In an aspect, rack114 may refer to the physical housing or platform for multiple serversor other network equipment. In an aspect, rack 114 may also refer to theunderlying network equipment. Each rack 114 may include one or moreservers 116. Server 116 may comprise general purpose computer hardwareor a computer. In an aspect, rack 114 may comprise a metal rack, andservers 116 of rack 114 may comprise blade servers that are physicallymounted in or on rack 114.

Each server 116 may include one or more network resources 118, asillustrated. Servers 116 may be communicatively linked together (notshown) in any combination or arrangement. For example, all servers 116within a given datacenter 102 or rack 114 may be in communication withone or more other servers 116. As another example, servers 116 indifferent racks 114 may be in communication with one or more otherservers 116 in one or more different racks 114. Additionally oralternatively, racks 114 may be communicatively coupled together (notshown) in any combination or arrangement.

The characteristics of each datacenter 102, rack 114, and server 116 maydiffer. For example, the number of racks 114 within two datacenters 102may vary. As another example, the number of servers 116 within two racks114 may vary. Additionally or alternatively, the type or number ofresources 118 within each server 116 may vary. In an aspect, rack 114may be used to group servers 116 with the same resource characteristics.In another aspect, servers 116 within the same rack 114 may havedifferent resource characteristics.

A single application may include many functional components, likenetwork functions. These components may have dependencies upon eachother and inter-communication patterns with certain quality of service(QoS) requirements, such as latency, locality, high availability, andsecurity. Consequently, placement decisions—that is, decisions on how(and where) to implement network functions within network 100—may bebased on the requirements of the network function and of the othernetwork functions instantiated on network 100, holistically.

For example, placement, that is, allocation and relocation ofinstantiations of network functions in network 100, may be based on oneor more resource requirements, affinity rules, diversity (oranti-affinity) rules, or pipe rules. A resource requirement may includea number or type of resource 118 that is required to instantiate anetwork function. An affinity rule may require that certaininstantiations or elements of a network function (e.g., its underlyingvirtual machines) be hosted together on the same server 116, rack 114,datacenter 102, or tier (e.g., edge tier 108). A diversity rule (e.g.,an anti-affinity rule) may require that certain instantiations orelements of a network function (e.g., its underlying virtual machines)be hosted on different servers 116, racks 114, datacenters 102, or tiers(e.g., edge tier 108 and regional tier 106). A pipe rule may requirethat a pairing of two elements of an instantiation of a network function(e.g., two virtual machines), or two instantiations, have a specificcommunication requirement (e.g., bandwidth or latency requirement).

FIG. 1C illustrates a system 120 that includes the relationships anorchestrator 122 uses to allocate and relocate resources 118 acrossnetwork 100. Instead of requiring multiple orchestrators to allocateresources at a datacenter 102 level or even at a regional level,orchestrator 122 uses a unified, rule-driven approach to the reservationof computing resources 118 for different levels of resiliency within alocal cloud environment (e.g., datacenter 102) and between multiple,geographically distributed cloud environments. Orchestrator 120 may usealgorithms to perform not only initial placement but also relocation andrebalancing of network functions for a service instance, based uponspecifications of resiliency needs and priority for the service.

A cloud infrastructure operator for the service may establish rules toprioritize allocation (or relocation) of resources for a serviceinstance. These rules may be input into a system via operator controlsystem 124. This allows for the cloud infrastructure operator tocustomize rules based on the specific requirements of the networkfunction or even the instantiation thereof. The rules can include, butare not limited to, local resiliency, which can address the reliabilityof resources 118 at datacenter 102. Additionally or alternatively therules can include a geographic resiliency. The geographic resiliency maybe related to the reliability of resources 118 at a specific geographiclocation, which can be addressed by selecting one or more datacenters102 in that geographic location from which to reserve resources 118. Forgeographic resiliency, datacenters 102 may be selected based on theirdiverse regions that provide a similar amount of control in each regionand a similar latency to managed edge network components or end devicesfrom a second datacenter 102 in the same region. For example, the rulesmay indicate a preference for a specific geographical area (e.g., withinthe state of Georgia) rather than requiring a preference of a specificdatacenter 102 (e.g., the XYZ datacenter).

The rules can also include control hierarchy requirements. For example,an instantiation may require a first set of resources 118 at a specifictier (e.g., global tier 104) and a second set of resources 118 thatinteract with the first set at a different tier (e.g., regional tier106).

The rules may prioritize the resource needs of an instantiation. Forexample, the resource needs may include some “needs” that are, inactuality, preferences. Orchestrator 120 may be tasked with weighingthese preferences against one another, both for the same instantiationand, weighing the preferences of different instantiations against oneanother. This can be facilitated by prioritizing a specificinstantiation over another, prioritizing a specific preference overanother, or a combination thereof. Orchestrator 122 may receiveinformation indicative of the rules set forth by the operator—that is,the “resource needs”—via operator controls 124.

Orchestrator 122 also receives information from an inventory 126 that iscurrently being used to obtain a better understanding of the content andavailability of resources within network 100. This may include serviceinstance specifications—that is, the resource needs of a givenapplication or network function—and the functionality of those networkfunctions. Combined with the rules received from operator controls 124,this inventory may provide a comprehensive picture of what aninstantiation will look like or how it will operate once implemented.

Orchestrator 122 may also receive network information from network 100.This includes the availability and configurations of datacenters 102,which can be as detailed as to indicate which resources 118 areavailable, reserved, in use, or offline (as a result of a failure or amaintenance operation).

Orchestrator may compile a resource map 128 based on this information.The resource map may represent relationships, specifications, andavailability of resources 118. The resource map may be used to keeptrack of the allocation of resources 118, for the purposes of relocatingresources 118 in the event of a failure, to accommodate newinstantiation requests, or to rebalance network 100.

FIG. 2A illustrates an exemplary method 200 by which orchestrator 122may allocate resources across multiple cloud computing environments fora network function. Variations of method 200 may achieve the samepurpose as method 200. Thus, not all steps illustrated in FIG. 2 ordescribed below are necessary for every implementation of method 200.Further, the following steps of method 200 are described using specificexamples, but none of these examples should be interpreted as the onlyor necessary implementation of such steps.

In exemplary method 200, at step 202 orchestrator 122 may receive anallocation request to instantiate a network function. This request mayinclude, or orchestrator 122 may otherwise obtain, informationindicative of the resource needs. As discussed above, resource needs maybe rules, which may be defined or input by a cloud operator. Theresource needs may include resiliency, including local resiliency and/orgeographic resiliency. Further information, including the specificationof the network function, may be obtained by orchestrator 122. Theresource needs may include a requirement or preference to beinstantiated in a specific datacenter 102 or a geographic region orhierarchal tier.

At step 204 orchestrator 122 may compute resource map 128. Computingresource map 128 may be performed as an ongoing function that includesupdating resource map 128 based on changes to network 100, includingallocation of resources 118 and unavailability of datacenters 102. Asdiscussed above, resource map 128 may include information indicating thedifferent tiers of network 100. Resource map 128 may indicate whichportions of network 100 are reserved or available, and it may includemore detailed information for unavailable resources 118, such as anindication of the function to which such resources 118 are alreadyassigned or may be offline.

At step 206 orchestrator 122 may compare the resource needs of theallocation request with resource map 128. This may result in identifyingone or more possible ways in which resources 118 can be allocated tosatisfy the resource needs of the network function. Multiple differentallocations may satisfy the resource needs. For example, a firstallocation may satisfy all of the requirements, but may not satisfy anoncritical preference, such as a geographic preference, while a secondallocation may satisfy both the mandatory and noncritical resourceneeds. In some circumstances, this comparing may result in a conclusionthat no placement would satisfy the resource needs of the allocationrequest. Depending upon priority of the request and the networkfunctions already instantiated, this could result in operator controls124 revising the rules, and resubmitting the request, or orchestrator122 rebalancing or relocating resources of other network functions.

In some instances, the resource needs are broadly defined so thatorchestrator 122 is tasked with interpreting the resource needs andcomparing those interpretations with resource map 128. For example, theresource needs may indicate a broad geographic area, and comparing theresource needs to resource map 128 may include identifying the differentdatacenters 102 within that geographic area and then identifying whichallocations are possible given the other resource needs and theavailability within those datacenters.

At step 208 orchestrator 122 may determine an allocation solution. Incircumstances in which only one configuration would satisfy theallocation needs, step 208 may simply mean selecting the allocationsolution identified in step 206. In circumstances in which multipleallocations are available, step 208 may weigh the different allocationsagainst one another, depending upon the rules set forth by operatorcontrols 124. For example, step 208 may include computing and comparingresiliency scores for different allocation solutions. These resiliencyscores may be based on the local resiliency and geographic resiliency ofresources 118. For example, a particular allocation solution thatsatisfies all resource needs—including noncritical needs—may be givenpreference, particularly if the allocation request is a high priorityrequest.

At step 210 orchestrator 122 may allocate (or cause to be allocated)resources 118. This allocation may comprise implementing the allocationsolution. Orchestrator 122 may update resource map 128 to reflect thatresources 118 allocated to the network function to satisfy theallocation request are in use. Real-time updating of resource map 128allows for dynamic service instance relocations, potentially decreasingdown time of network functions that can occur as the result of a networkfailure. In the same vein, resource map 128 may be updated to includepredictive heat maps of the free resources 118 that would be availablein the event of a resource failure. This information may be used topreemptively identify relocation solutions that can be implemented inthe case of an actual resource failure.

Orchestrator 122 may also provide other, related functionality thatallows for relocation of a network function to different resources 128,such as in the event of a failure of those resources 128, a morepreferred allocation becoming available, or network rebalancing.Orchestrator 122 may allow an administrator to relocate serviceinstances of network functions to other datacenters 102 in compliancewith specifications of those network functions from inventory 126.Relocation can be prioritized, so that orchestrator 122 prioritizesrelocation of high priority network functions, like those related toemergency communications, over other, lower priority network functions.

Further, orchestrator 128 may also react to the creation or availabilityof new datacenters 102 (or new resources 118). For example, orchestrator128 may assess the resource needs of services instances of networkfunctions—particularly to identify different assignments that can moreefficiently use resources 118. That is, the introduction of newdatacenters 102 or resources 118 may provide more optimized uses ofresources 118 that may be implemented by reassigning resources 118 tothe different service instances currently in use.

Finally, the functionality of orchestrator 122 may incorporate feedbacksystems that allow optimization of its operations, including the use ofmachine learning to react to automatic resiliency events or to detectand implement rebalancing solutions across network 100. Analytics of theprobabilities and characteristics of relocation requests due toautomatic resiliency events, or approved rebalancing actions, may beused to anticipate network changes and to proactively relocate networkfunctions.

FIG. 3 is a block diagram of network device 300 that may be connected toor comprise a component of network 100 or system 120. For example,network device 300 may implement one or more portions of method 200 forallocation of resources 118. Network device 300 may comprise hardware ora combination of hardware and software. The functionality to facilitatetelecommunications via a telecommunications network may reside in one orcombination of network devices 300. Network device 300 depicted in FIG.3 may represent or perform functionality of an appropriate networkdevice 300, or combination of network devices 300, such as, for example,a component or various components of a cellular broadcast systemwireless network, a processor, a server, a gateway, a node, a mobileswitching center (MSC), a short message service center (SMSC), an ALFS,a gateway mobile location center (GMLC), a radio access network (RAN), aserving mobile location center (SMLC), or the like, or any appropriatecombination thereof. It is emphasized that the block diagram depicted inFIG. 3 is exemplary and not intended to imply a limitation to a specificimplementation or configuration. Thus, network device 300 may beimplemented in a single device or multiple devices (e.g., single serveror multiple servers, single gateway or multiple gateways, singlecontroller or multiple controllers). Multiple network entities may bedistributed or centrally located. Multiple network entities maycommunicate wirelessly, via hard wire, or any appropriate combinationthereof.

Network device 300 may comprise a processor 302 and a memory 304 coupledto processor 302. Memory 304 may contain executable instructions that,when executed by processor 302, cause processor 302 to effectuateoperations associated with mapping wireless signal strength. As evidentfrom the description herein, network device 300 is not to be construedas software per se.

In addition to processor 302 and memory 304, network device 300 mayinclude an input/output system 306. Processor 302, memory 304, andinput/output system 306 may be coupled together (coupling not shown inFIG. 3) to allow communications therebetween. Each portion of networkdevice 300 may comprise circuitry for performing functions associatedwith each respective portion. Thus, each portion may comprise hardware,or a combination of hardware and software. Accordingly, each portion ofnetwork device 300 is not to be construed as software per se.Input/output system 306 may be capable of receiving or providinginformation from or to a communications device or other network entitiesconfigured for telecommunications. For example input/output system 306may include a wireless communications (e.g., 3G/4G/GPS) card.Input/output system 306 may be capable of receiving or sending videoinformation, audio information, control information, image information,data, or any combination thereof. Input/output system 306 may be capableof transferring information with network device 300. In variousconfigurations, input/output system 306 may receive or provideinformation via any appropriate means, such as, for example, opticalmeans (e.g., infrared), electromagnetic means (e.g., RF, Wi-Fi,Bluetooth®, ZigBee®), acoustic means (e.g., speaker, microphone,ultrasonic receiver, ultrasonic transmitter), or a combination thereof.In an example configuration, input/output system 306 may comprise aWi-Fi finder, a two-way GPS chipset or equivalent, or the like, or acombination thereof.

Input/output system 306 of network device 300 also may contain acommunication connection 308 that allows network device 300 tocommunicate with other devices, network entities, or the like.Communication connection 308 may comprise communication media.Communication media typically embody computer-readable instructions,data structures, program modules or other data in a modulated datasignal such as a carrier wave or other transport mechanism and includesany information delivery media. By way of example, and not limitation,communication media may include wired media such as a wired network ordirect-wired connection, or wireless media such as acoustic, RF,infrared, or other wireless media. The term computer-readable media asused herein includes both storage media and communication media.Input/output system 306 also may include an input device 310 such askeyboard, mouse, pen, voice input device, or touch input device.Input/output system 306 may also include an output device 312, such as adisplay, speakers, or a printer.

Processor 302 may be capable of performing functions associated withtelecommunications, such as functions for processing broadcast messages,as described herein. For example, processor 302 may be capable of, inconjunction with any other portion of network device 300, determining atype of broadcast message and acting according to the broadcast messagetype or content, as described herein.

Memory 304 of network device 300 may comprise a storage medium having aconcrete, tangible, physical structure. As is known, a signal does nothave a concrete, tangible, physical structure. Memory 304, as well asany computer-readable storage medium described herein, is not to beconstrued as a signal. Memory 304, as well as any computer-readablestorage medium described herein, is not to be construed as a transientsignal. Memory 304, as well as any computer-readable storage mediumdescribed herein, is not to be construed as a propagating signal. Memory304, as well as any computer-readable storage medium described herein,is to be construed as an article of manufacture.

Memory 304 may store any information utilized in conjunction withtelecommunications. Depending upon the exact configuration or type ofprocessor, memory 304 may include a volatile storage 314 (such as sometypes of RAM), a nonvolatile storage 316 (such as ROM, flash memory), ora combination thereof. Memory 304 may include additional storage (e.g.,a removable storage 318 or a nonremovable storage 320) including, forexample, tape, flash memory, smart cards, CD-ROM, DVD, or other opticalstorage, magnetic cassettes, magnetic tape, magnetic disk storage orother magnetic storage devices, USB-compatible memory, or any othermedium that can be used to store information and that can be accessed bynetwork device 300. Memory 304 may comprise executable instructionsthat, when executed by processor 302, cause processor 302 to effectuateoperations to map signal strengths in an area of interest.

FIG. 4 illustrates a functional block diagram depicting one example ofan LTE-EPS network architecture 400 that may be at least partiallyimplemented as using virtualized functions. Network architecture 400disclosed herein is referred to as a modified LTE-EPS architecture 400to distinguish it from a traditional LTE-EPS architecture.

An example modified LTE-EPS architecture 400 is based at least in parton standards developed by the 3rd Generation Partnership Project (3GPP),with information available at www.3gpp.org. LTE-EPS network architecture400 may include an access network 402, a core network 404, e.g., an EPCor Common BackBone (CBB) and one or more external networks 406,sometimes referred to as PDN or peer entities. Different externalnetworks 406 can be distinguished from each other by a respectivenetwork identifier, e.g., a label according to DNS naming conventionsdescribing an access point to the PDN. Such labels can be referred to asAccess Point Names (APN). External networks 406 can include one or moretrusted and non-trusted external networks such as an internet protocol(IP) network 408, an IP multimedia subsystem (IMS) network 410, andother networks 412, such as a service network, a corporate network, orthe like. In an aspect, access network 402, core network 404, orexternal network 405 may include or communicate with network 100.

Access network 402 can include an LTE network architecture sometimesreferred to as Evolved Universal mobile Telecommunication systemTerrestrial Radio Access (E UTRA) and evolved UMTS Terrestrial RadioAccess Network (E-UTRAN). Broadly, access network 402 can include one ormore communication devices, commonly referred to as UE 414, and one ormore wireless access nodes, or base stations 416 a, 416 b. Duringnetwork operations, at least one base station 416 communicates directlywith UE 414. Base station 416 can be an evolved Node B (e-NodeB), withwhich UE 414 communicates over the air and wirelessly. UEs 414 caninclude, without limitation, wireless devices, e.g., satellitecommunication systems, portable digital assistants (PDAs), laptopcomputers, tablet devices and other mobile devices (e.g., cellulartelephones, smart appliances, and so on). UEs 414 can connect to eNBs416 when UE 414 is within range according to a corresponding wirelesscommunication technology.

UE 414 generally runs one or more applications that engage in a transferof packets between UE 414 and one or more external networks 406. Suchpacket transfers can include one of downlink packet transfers fromexternal network 406 to UE 414, uplink packet transfers from UE 414 toexternal network 406 or combinations of uplink and downlink packettransfers. Applications can include, without limitation, web browsing,VoIP, streaming media and the like. Each application can pose differentQuality of Service (QoS) requirements on a respective packet transfer.Different packet transfers can be served by different bearers withincore network 404, e.g., according to parameters, such as the QoS.

Core network 404 uses a concept of bearers, e.g., EPS bearers, to routepackets, e.g., IP traffic, between a particular gateway in core network404 and UE 414. A bearer refers generally to an IP packet flow with adefined QoS between the particular gateway and UE 414. Access network402, e.g., E UTRAN, and core network 404 together set up and releasebearers as required by the various applications. Bearers can beclassified in at least two different categories: (i) minimum guaranteedbit rate bearers, e.g., for applications, such as VoIP; and (ii)non-guaranteed bit rate bearers that do not require guarantee bit rate,e.g., for applications, such as web browsing.

In one embodiment, the core network 404 includes various networkentities, such as MME 418, SGW 420, Home Subscriber Server (HSS) 422,Policy and Charging Rules Function (PCRF) 424 and PGW 426. In oneembodiment, MME 418 comprises a control node performing a controlsignaling between various equipment and devices in access network 402and core network 404. The protocols running between UE 414 and corenetwork 404 are generally known as Non-Access Stratum (NAS) protocols.

For illustration purposes only, the terms MME 418, SGW 420, HSS 422 andPGW 426, and so on, can be server devices, but may be referred to in thesubject disclosure without the word “server.” It is also understood thatany form of such servers can operate in a device, system, component, orother form of centralized or distributed hardware and software. It isfurther noted that these terms and other terms such as bearer pathsand/or interfaces are terms that can include features, methodologies,and/or fields that may be described in whole or in part by standardsbodies such as the 3GPP. It is further noted that some or allembodiments of the subject disclosure may in whole or in part modify,supplement, or otherwise supersede final or proposed standards publishedand promulgated by 3GPP.

According to traditional implementations of LTE-EPS architectures, SGW420 routes and forwards all user data packets. SGW 420 also acts as amobility anchor for user plane operation during handovers between basestations, e.g., during a handover from first eNB 416 a to second eNB 416b as may be the result of UE 414 moving from one area of coverage, e.g.,cell, to another. SGW 420 can also terminate a downlink data path, e.g.,from external network 406 to UE 414 in an idle state, and trigger apaging operation when downlink data arrives for UE 414. SGW 420 can alsobe configured to manage and store a context for UE 414, e.g., includingone or more of parameters of the IP bearer service and network internalrouting information. In addition, SGW 420 can perform administrativefunctions, e.g., in a visited network, such as collecting informationfor charging (e.g., the volume of data sent to or received from theuser), and/or replicate user traffic, e.g., to support a lawfulinterception. SGW 420 also serves as the mobility anchor forinterworking with other 3GPP technologies such as universal mobiletelecommunication system (UMTS).

At any given time, UE 414 is generally in one of three different states:detached, idle, or active. The detached state is typically a transitorystate in which UE 414 is powered on but is engaged in a process ofsearching and registering with network 402. In the active state, UE 414is registered with access network 402 and has established a wirelessconnection, e.g., radio resource control (RRC) connection, with eNB 416.Whether UE 414 is in an active state can depend on the state of a packetdata session, and whether there is an active packet data session. In theidle state, UE 414 is generally in a power conservation state in whichUE 414 typically does not communicate packets. When UE 414 is idle, SGW420 can terminate a downlink data path, e.g., from one peer entity, andtriggers paging of UE 414 when data arrives for UE 414. If UE 414responds to the page, SGW 420 can forward the IP packet to eNB 416 a.

HSS 422 can manage subscription-related information for a user of UE414. For example, tHSS 422 can store information such as authorizationof the user, security requirements for the user, quality of service(QoS) requirements for the user, etc. HSS 422 can also hold informationabout external networks 406 to which the user can connect, e.g., in theform of an APN of external networks 406. For example, MME 418 cancommunicate with HSS 422 to determine if UE 414 is authorized toestablish a call, e.g., a voice over IP (VoIP) call before the call isestablished.

PCRF 424 can perform QoS management functions and policy control. PCRF424 is responsible for policy control decision-making, as well as forcontrolling the flow-based charging functionalities in a policy controlenforcement function (PCEF), which resides in PGW 426. PCRF 424 providesthe QoS authorization, e.g., QoS class identifier and bit rates thatdecide how a certain data flow will be treated in the PCEF and ensuresthat this is in accordance with the user's subscription profile.

PGW 426 can provide connectivity between the UE 414 and one or more ofthe external networks 406. In illustrative network architecture 400, PGW426 can be responsible for IP address allocation for UE 414, as well asone or more of QoS enforcement and flow-based charging, e.g., accordingto rules from the PCRF 424. PGW 426 is also typically responsible forfiltering downlink user IP packets into the different QoS-based bearers.In at least some embodiments, such filtering can be performed based ontraffic flow templates. PGW 426 can also perform QoS enforcement, e.g.,for guaranteed bit rate bearers. PGW 426 also serves as a mobilityanchor for interworking with non-3GPP technologies such as CDMA2000.

Within access network 402 and core network 404 there may be variousbearer paths/interfaces, e.g., represented by solid lines 428 and 430.Some of the bearer paths can be referred to by a specific label. Forexample, solid line 428 can be considered an S1-U bearer and solid line432 can be considered an S5/S8 bearer according to LTE-EPS architecturestandards. Without limitation, reference to various interfaces, such asS1, X2, S5, S8, S11 refer to EPS interfaces. In some instances, suchinterface designations are combined with a suffix, e.g., a “U” or a “C”to signify whether the interface relates to a “User plane” or a “Controlplane.” In addition, the core network 404 can include various signalingbearer paths/interfaces, e.g., control plane paths/interfacesrepresented by dashed lines 430, 434, 436, and 438. Some of thesignaling bearer paths may be referred to by a specific label. Forexample, dashed line 430 can be considered as an S1-MME signalingbearer, dashed line 434 can be considered as an S11 signaling bearer anddashed line 436 can be considered as an S6a signaling bearer, e.g.,according to LTE-EPS architecture standards. The above bearer paths andsignaling bearer paths are only illustrated as examples and it should benoted that additional bearer paths and signaling bearer paths may existthat are not illustrated.

Also shown is a novel user plane path/interface, referred to as theS1-U+ interface 466. In the illustrative example, the S1-U+ user planeinterface extends between the eNB 416 a and PGW 426. Notably, S1-U+path/interface does not include SGW 420, a node that is otherwiseinstrumental in configuring and/or managing packet forwarding betweeneNB 416 a and one or more external networks 406 by way of PGW 426. Asdisclosed herein, the S1-U+ path/interface facilitates autonomouslearning of peer transport layer addresses by one or more of the networknodes to facilitate a self-configuring of the packet forwarding path. Inparticular, such self-configuring can be accomplished during handoversin most scenarios so as to reduce any extra signaling load on the S/PGWs420, 426 due to excessive handover events.

In some embodiments, PGW 426 is coupled to storage device 440, shown inphantom. Storage device 440 can be integral to one of the network nodes,such as PGW 426, for example, in the form of internal memory and/or diskdrive. It is understood that storage device 440 can include registerssuitable for storing address values. Alternatively or in addition,storage device 440 can be separate from PGW 426, for example, as anexternal hard drive, a flash drive, and/or network storage.

Storage device 440 selectively stores one or more values relevant to theforwarding of packet data. For example, storage device 440 can storeidentities and/or addresses of network entities, such as any of networknodes 418, 420, 422, 424, and 426, eNBs 416 and/or UE 414. In theillustrative example, storage device 440 includes a first storagelocation 442 and a second storage location 444. First storage location442 can be dedicated to storing a Currently Used Downlink address value442. Likewise, second storage location 444 can be dedicated to storing aDefault Downlink Forwarding address value 444. PGW 426 can read and/orwrite values into either of storage locations 442, 444, for example,managing Currently Used Downlink Forwarding address value 442 andDefault Downlink Forwarding address value 444 as disclosed herein.

In some embodiments, the Default Downlink Forwarding address for eachEPS bearer is the SGW S5-U address for each EPS Bearer. The CurrentlyUsed Downlink Forwarding address” for each EPS bearer in PGW 426 can beset every time when PGW 426 receives an uplink packet, e.g., a GTP-Uuplink packet, with a new source address for a corresponding EPS bearer.When UE 414 is in an idle state, the “Current Used Downlink Forwardingaddress” field for each EPS bearer of UE 414 can be set to a “null” orother suitable value.

In some embodiments, the Default Downlink Forwarding address is onlyupdated when PGW 426 receives a new SGW S5-U address in a predeterminedmessage or messages. For example, the Default Downlink Forwardingaddress is only updated when PGW 426 receives one of a Create SessionRequest, Modify Bearer Request and Create Bearer Response messages fromSGW 420.

As values 442, 444 can be maintained and otherwise manipulated on a perbearer basis, it is understood that the storage locations can take theform of tables, spreadsheets, lists, and/or other data structuresgenerally well understood and suitable for maintaining and/or otherwisemanipulate forwarding addresses on a per bearer basis.

It should be noted that access network 402 and core network 404 areillustrated in a simplified block diagram in FIG. 4. In other words,either or both of access network 402 and the core network 404 caninclude additional network elements that are not shown, such as variousrouters, switches and controllers. In addition, although FIG. 4illustrates only a single one of each of the various network elements,it should be noted that access network 402 and core network 404 caninclude any number of the various network elements. For example, corenetwork 404 can include a pool (i.e., more than one) of MMEs 418, SGWs420 or PGWs 426.

In the illustrative example, data traversing a network path between UE414, eNB 416 a, SGW 420, PGW 426 and external network 406 may beconsidered to constitute data transferred according to an end-to-end IPservice. However, for the present disclosure, to properly performestablishment management in LTE-EPS network architecture 400, the corenetwork, data bearer portion of the end-to-end IP service is analyzed.

An establishment may be defined herein as a connection set up requestbetween any two elements within LTE-EPS network architecture 400. Theconnection set up request may be for user data or for signaling. Afailed establishment may be defined as a connection set up request thatwas unsuccessful. A successful establishment may be defined as aconnection set up request that was successful.

In one embodiment, a data bearer portion comprises a first portion(e.g., a data radio bearer 446) between UE 414 and eNB 416 a, a secondportion (e.g., an S1 data bearer 428) between eNB 416 a and SGW 420, anda third portion (e.g., an S5/S8 bearer 432) between SGW 420 and PGW 426.Various signaling bearer portions are also illustrated in FIG. 4. Forexample, a first signaling portion (e.g., a signaling radio bearer 448)between UE 414 and eNB 416 a, and a second signaling portion (e.g., S1signaling bearer 430) between eNB 416 a and MME 418.

In at least some embodiments, the data bearer can include tunneling,e.g., IP tunneling, by which data packets can be forwarded in anencapsulated manner, between tunnel endpoints. Tunnels, or tunnelconnections can be identified in one or more nodes of network 100, e.g.,by one or more of tunnel endpoint identifiers, an IP address and a userdatagram protocol port number. Within a particular tunnel connection,payloads, e.g., packet data, which may or may not include protocolrelated information, are forwarded between tunnel endpoints.

An example of first tunnel solution 450 includes a first tunnel 452 abetween two tunnel endpoints 454 a and 456 a, and a second tunnel 452 bbetween two tunnel endpoints 454 b and 456 b. In the illustrativeexample, first tunnel 452 a is established between eNB 416 a and SGW420. Accordingly, first tunnel 452 a includes a first tunnel endpoint454 a corresponding to an S1-U address of eNB 416 a (referred to hereinas the eNB S1-U address), and second tunnel endpoint 456 a correspondingto an S1-U address of SGW 420 (referred to herein as the SGW S1-Uaddress). Likewise, second tunnel 452 b includes first tunnel endpoint454 b corresponding to an S5-U address of SGW 420 (referred to herein asthe SGW S5-U address), and second tunnel endpoint 456 b corresponding toan S5-U address of PGW 426 (referred to herein as the PGW S5-U address).

In at least some embodiments, first tunnel solution 450 is referred toas a two tunnel solution, e.g., according to the GPRS Tunneling ProtocolUser Plane (GTPv1-U based), as described in 3GPP specification TS29.281, incorporated herein in its entirety. It is understood that oneor more tunnels are permitted between each set of tunnel end points. Forexample, each subscriber can have one or more tunnels, e.g., one foreach PDP context that they have active, as well as possibly havingseparate tunnels for specific connections with different quality ofservice requirements, and so on.

An example of second tunnel solution 458 includes a single or directtunnel 460 between tunnel endpoints 462 and 464. In the illustrativeexample, direct tunnel 460 is established between eNB 416 a and PGW 426,without subjecting packet transfers to processing related to SGW 420.Accordingly, direct tunnel 460 includes first tunnel endpoint 462corresponding to the eNB S1-U address, and second tunnel endpoint 464corresponding to the PGW S5-U address. Packet data received at eitherend can be encapsulated into a payload and directed to the correspondingaddress of the other end of the tunnel. Such direct tunneling avoidsprocessing, e.g., by SGW 420 that would otherwise relay packets betweenthe same two endpoints, e.g., according to a protocol, such as the GTP-Uprotocol.

In some scenarios, direct tunneling solution 458 can forward user planedata packets between eNB 416 a and PGW 426, by way of SGW 420. That is,SGW 420 can serve a relay function, by relaying packets between twotunnel endpoints 416 a, 426. In other scenarios, direct tunnelingsolution 458 can forward user data packets between eNB 416 a and PGW426, by way of the S1 U+ interface, thereby bypassing SGW 420.

Generally, UE 414 can have one or more bearers at any one time. Thenumber and types of bearers can depend on applications, defaultrequirements, and so on. It is understood that the techniques disclosedherein, including the configuration, management and use of varioustunnel solutions 450, 458, can be applied to the bearers on anindividual bases. That is, if user data packets of one bearer, say abearer associated with a VoIP service of UE 414, then the forwarding ofall packets of that bearer are handled in a similar manner. Continuingwith this example, the same UE 414 can have another bearer associatedwith it through the same eNB 416 a. This other bearer, for example, canbe associated with a relatively low rate data session forwarding userdata packets through core network 404 simultaneously with the firstbearer. Likewise, the user data packets of the other bearer are alsohandled in a similar manner, without necessarily following a forwardingpath or solution of the first bearer. Thus, one of the bearers may beforwarded through direct tunnel 458; whereas, another one of the bearersmay be forwarded through a two-tunnel solution 450.

FIG. 5 depicts an exemplary diagrammatic representation of a machine inthe form of a computer system 500 within which a set of instructions,when executed, may cause the machine to perform any one or more of themethods described above. One or more instances of the machine canoperate, for example, as processor 302, UE 414, eNB 416, MME 418, SGW420, HSS 422, PCRF 424, PGW 426 and other devices of FIGS. 1, 2, and 4.In some embodiments, the machine may be connected (e.g., using a network502) to other machines. In a networked deployment, the machine mayoperate in the capacity of a server or a client user machine in aserver-client user network environment, or as a peer machine in apeer-to-peer (or distributed) network environment.

The machine may comprise a server computer, a client user computer, apersonal computer (PC), a tablet, a smart phone, a laptop computer, adesktop computer, a control system, a network router, switch or bridge,or any machine capable of executing a set of instructions (sequential orotherwise) that specify actions to be taken by that machine. It will beunderstood that a communication device of the subject disclosureincludes broadly any electronic device that provides voice, video ordata communication. Further, while a single machine is illustrated, theterm “machine” shall also be taken to include any collection of machinesthat individually or jointly execute a set (or multiple sets) ofinstructions to perform any one or more of the methods discussed herein.

Computer system 500 may include a processor (or controller) 504 (e.g., acentral processing unit (CPU)), a graphics processing unit (GPU, orboth), a main memory 506 and a static memory 508, which communicate witheach other via a bus 510. The computer system 500 may further include adisplay unit 512 (e.g., a liquid crystal display (LCD), a flat panel, ora solid state display). Computer system 500 may include an input device514 (e.g., a keyboard), a cursor control device 516 (e.g., a mouse), adisk drive unit 518, a signal generation device 520 (e.g., a speaker orremote control) and a network interface device 522. In distributedenvironments, the embodiments described in the subject disclosure can beadapted to utilize multiple display units 512 controlled by two or morecomputer systems 500. In this configuration, presentations described bythe subject disclosure may in part be shown in a first of display units512, while the remaining portion is presented in a second of displayunits 512.

The disk drive unit 518 may include a tangible computer-readable storagemedium 524 on which is stored one or more sets of instructions (e.g.,software 526) embodying any one or more of the methods or functionsdescribed herein, including those methods illustrated above.Instructions 526 may also reside, completely or at least partially,within main memory 506, static memory 508, or within processor 504during execution thereof by the computer system 500. Main memory 506 andprocessor 504 also may constitute tangible computer-readable storagemedia.

As shown in FIG. 6, telecommunication system 600 may include wirelesstransmit/receive units (WTRUs) 602, a RAN 604, a core network 606, apublic switched telephone network (PSTN) 608, the Internet 610, or othernetworks 612, though it will be appreciated that the disclosed examplescontemplate any number of WTRUs, base stations, networks, or networkelements. Each WTRU 602 may be any type of device configured to operateor communicate in a wireless environment. For example, a WTRU maycomprise a mobile device, network device 300, or the like, or anycombination thereof. By way of example, WTRUs 602 may be configured totransmit or receive wireless signals and may include a UE, a mobilestation, a mobile device, a fixed or mobile subscriber unit, a pager, acellular telephone, a PDA, a smartphone, a laptop, a netbook, a personalcomputer, a wireless sensor, consumer electronics, or the like. WTRUs602 may be configured to transmit or receive wireless signals over anair interface 614.

Telecommunication system 600 may also include one or more base stations616. Each of base stations 616 may be any type of device configured towirelessly interface with at least one of the WTRUs 602 to facilitateaccess to one or more communication networks, such as core network 606,PTSN 608, Internet 610, or other networks 612. By way of example, basestations 616 may be a base transceiver station (BTS), a Node-B, an eNodeB, a Home Node B, a Home eNode B, a site controller, an access point(AP), a wireless router, or the like. While base stations 616 are eachdepicted as a single element, it will be appreciated that base stations616 may include any number of interconnected base stations or networkelements.

RAN 604 may include one or more base stations 616, along with othernetwork elements (not shown), such as a base station controller (BSC), aradio network controller (RNC), or relay nodes. One or more basestations 616 may be configured to transmit or receive wireless signalswithin a particular geographic region, which may be referred to as acell (not shown). The cell may further be divided into cell sectors. Forexample, the cell associated with base station 616 may be divided intothree sectors such that base station 616 may include three transceivers:one for each sector of the cell. In another example, base station 616may employ multiple-input multiple-output (MIMO) technology and,therefore, may utilize multiple transceivers for each sector of thecell.

Base stations 616 may communicate with one or more of WTRUs 602 over airinterface 614, which may be any suitable wireless communication link(e.g., RF, microwave, infrared (IR), ultraviolet (UV), or visiblelight). Air interface 614 may be established using any suitable radioaccess technology (RAT).

More specifically, as noted above, telecommunication system 600 may be amultiple access system and may employ one or more channel accessschemes, such as CDMA, TDMA, FDMA, OFDMA, SC-FDMA, or the like. Forexample, base station 616 in RAN 604 and WTRUs 602 connected to RAN 604may implement a radio technology such as Universal MobileTelecommunications System (UMTS) Terrestrial Radio Access (UTRA) thatmay establish air interface 614 using wideband CDMA (WCDMA). WCDMA mayinclude communication protocols, such as High-Speed Packet Access (HSPA)or Evolved HSPA (HSPA+). HSPA may include High-Speed Downlink PacketAccess (HSDPA) or High-Speed Uplink Packet Access (HSUPA).

As another example base station 616 and WTRUs 602 that are connected toRAN 604 may implement a radio technology such as Evolved UMTSTerrestrial Radio Access (E-UTRA), which may establish air interface 614using LTE or LTE-Advanced (LTE-A).

Optionally base station 616 and WTRUs 602 connected to RAN 604 mayimplement radio technologies such as IEEE 602.16 (i.e., WorldwideInteroperability for Microwave Access (WiMAX)), CDMA2000, CDMA2000 1X,CDMA2000 EV-DO, Interim Standard 2000 (IS-2000), Interim Standard 95(IS-95), Interim Standard 856 (IS-856), GSM, Enhanced Data rates for GSMEvolution (EDGE), GSM EDGE (GERAN), or the like.

Base station 616 may be a wireless router, Home Node B, Home eNode B, oraccess point, for example, and may utilize any suitable RAT forfacilitating wireless connectivity in a localized area, such as a placeof business, a home, a vehicle, a campus, or the like. For example, basestation 616 and associated WTRUs 602 may implement a radio technologysuch as IEEE 602.11 to establish a wireless local area network (WLAN).As another example, base station 616 and associated WTRUs 602 mayimplement a radio technology such as IEEE 602.15 to establish a wirelesspersonal area network (WPAN). In yet another example, base station 616and associated WTRUs 602 may utilize a cellular-based RAT (e.g., WCDMA,CDMA2000, GSM, LTE, LTE-A, etc.) to establish a picocell or femtocell.As shown in FIG. 6, base station 616 may have a direct connection toInternet 610. Thus, base station 616 may not be required to accessInternet 610 via core network 606.

RAN 604 may be in communication with core network 606, which may be anytype of network configured to provide voice, data, applications, and/orvoice over internet protocol (VoIP) services to one or more WTRUs 602.For example, core network 606 may provide call control, billingservices, mobile location-based services, pre-paid calling, Internetconnectivity, video distribution or high-level security functions, suchas user authentication. Although not shown in FIG. 6, it will beappreciated that RAN 604 or core network 606 may be in direct orindirect communication with other RANs that employ the same RAT as RAN604 or a different RAT. For example, in addition to being connected toRAN 604, which may be utilizing an E-UTRA radio technology, core network606 may also be in communication with another RAN (not shown) employinga GSM radio technology.

Core network 606 may also serve as a gateway for WTRUs 602 to accessPSTN 608, Internet 610, or other networks 612. PSTN 608 may includecircuit-switched telephone networks that provide plain old telephoneservice (POTS). For LTE core networks, core network 606 may use IMS core614 to provide access to PSTN 608. Internet 610 may include a globalsystem of interconnected computer networks or devices that use commoncommunication protocols, such as the transmission control protocol(TCP), user datagram protocol (UDP), or IP in the TCP/IP internetprotocol suite. Other networks 612 may include wired or wirelesscommunications networks owned or operated by other service providers.For example, other networks 612 may include another core networkconnected to one or more RANs, which may employ the same RAT as RAN 604or a different RAT.

Some or all WTRUs 602 in telecommunication system 600 may includemulti-mode capabilities. That is, WTRUs 602 may include multipletransceivers for communicating with different wireless networks overdifferent wireless links. For example, one or more WTRUs 602 may beconfigured to communicate with base station 616, which may employ acellular-based radio technology, and with base station 616, which mayemploy an IEEE 802 radio technology.

FIG. 7 is an example system 700 including RAN 604 and core network 606.As noted above, RAN 604 may employ an E-UTRA radio technology tocommunicate with WTRUs 602 over air interface 614. RAN 604 may also bein communication with core network 606.

RAN 604 may include any number of eNode-Bs 702 while remainingconsistent with the disclosed technology. One or more eNode-Bs 702 mayinclude one or more transceivers for communicating with the WTRUs 602over air interface 614. Optionally, eNode-Bs 702 may implement MIMOtechnology. Thus, one of eNode-Bs 702, for example, may use multipleantennas to transmit wireless signals to, or receive wireless signalsfrom, one of WTRUs 602.

Each of eNode-Bs 702 may be associated with a particular cell (notshown) and may be configured to handle radio resource managementdecisions, handover decisions, scheduling of users in the uplink ordownlink, or the like. As shown in FIG. 7 eNode-Bs 702 may communicatewith one another over an X2 interface.

Core network 606 shown in FIG. 7 may include a mobility managementgateway or entity (MME) 704, a serving gateway 706, or a packet datanetwork (PDN) gateway 708. While each of the foregoing elements aredepicted as part of core network 606, it will be appreciated that anyone of these elements may be owned or operated by an entity other thanthe core network operator.

MME 704 may be connected to each of eNode-Bs 702 in RAN 604 via an S1interface and may serve as a control node. For example, MME 704 may beresponsible for authenticating users of WTRUs 602, bearer activation ordeactivation, selecting a particular serving gateway during an initialattach of WTRUs 602, or the like. MME 704 may also provide a controlplane function for switching between RAN 604 and other RANs (not shown)that employ other radio technologies, such as GSM or WCDMA.

Serving gateway 706 may be connected to each of eNode-Bs 702 in RAN 604via the S1 interface. Serving gateway 706 may generally route or forwarduser data packets to or from the WTRUs 602. Serving gateway 706 may alsoperform other functions, such as anchoring user planes duringinter-eNode B handovers, triggering paging when downlink data isavailable for WTRUs 602, managing or storing contexts of WTRUs 602, orthe like.

Serving gateway 706 may also be connected to PDN gateway 708, which mayprovide WTRUs 602 with access to packet-switched networks, such asInternet 610, to facilitate communications between WTRUs 602 andIP-enabled devices.

Core network 606 may facilitate communications with other networks. Forexample, core network 606 may provide WTRUs 602 with access tocircuit-switched networks, such as PSTN 608, such as through IMS core614, to facilitate communications between WTRUs 602 and traditionalland-line communications devices. In addition, core network 606 mayprovide the WTRUs 602 with access to other networks 612, which mayinclude other wired or wireless networks that are owned or operated byother service providers.

The invention claimed is:
 1. A cloud orchestrator comprising: a networkconnection for connecting to a cloud network; a processorcommunicatively coupled to the network connection; and memory storinginstructions that cause the processor to effectuate operations, theoperations comprising: receiving a request to allocate an instantiationof a network function, wherein the request comprises informationindicative of resource requirements of the instantiation, wherein theresource requirements comprise at least one resiliency requirement, andwherein the at least one resiliency requirement comprises selecting oneor more datacenters that provide a similar amount of control in a regionand each of the one or more datacenters being associated with a similarlatency to managed edgenetwork components in the region; computing aresource map of the cloud network, the resource map comprising aglobaltier, and a regional tier; comparing the resource requirementswith the resource map to determine a plurality ofallocation solutions;computing a resiliency score for each of the plurality of allocationsolutions, wherein the resiliency score is based on a local resiliencyand a geographic resiliency of resources in the plurality of allocationsolutions; based on the resiliency score and the resiliency requirement,allocating resources to the instantiation, wherein the resourcescomprise a first resource of the global tier and a second resource ofthe regional tier; responsive to the allocating the resources to theinstantiation, periodically: updating the resource map of the cloudnetwork to generate a predictive heat map of free resources of the cloudnetwork; and determining a plurality of potential reallocation solutionsfor mitigating potential resource failures affecting the networkfunction according to the predictive heat map of free resources of thecloud network; detecting a first resource failure affecting the networkfunction; and implementing a first reallocation solution of theplurality of potential reallocation solutions responsive to thedetecting the first resource failure affecting the network function. 2.The cloud orchestrator of claim 1, wherein the resource requirementscomprise a geographic preference and selecting the second resource isbased on the geographic preference.
 3. The cloud orchestrator of claim1, wherein the resiliency requirement comprises a local resiliency and ageographic resiliency.
 4. The cloud orchestrator of claim 1, wherein theresource requirements comprise acontrol hierarchy that is satisfied byallocating the first resource from the global tier.
 5. The cloudorchestrator of claim 1, wherein the information is further indicativeof a geographic preference and the allocation solution is further basedon comparing the geographic preference with the resource map.
 6. Thecloud orchestrator of claim 1, the operations further comprising:determining that a new datacenter has been created within the regionaltier; and controlling relocation of the instantiation from the secondresource to a third resource of the new datacenter.
 7. The cloudorchestrator of claim 1, wherein the resource map further comprises anedge tier, wherein a first latency of the edge tier is lower than asecond latency of the regional tier.
 8. A method comprising: receiving,by a processing system including a processor, a request to allocate aninstantiation of a network function, wherein the request comprisesinformation indicative of resource requirements of the instantiation,wherein the resource requirements comprises at least one resiliencyrequirement, and wherein the at least one resiliency requirementcomprises selecting one or more datacenters that provide a similaramount of control in a region and each of the one ormore datacentersbeing associated with a similar latency to managed edge networkcomponents in the region; computing, by the processing system, aresource map of a cloud network comprising a global tier and a regionaltier; comparing, by the processing system, the resource requirementswith the resource map to determine a plurality of allocation solutions;computing, by the processing system, a resiliency score for each of theplurality of allocation solutions, wherein the resiliency score is basedon a local resiliency and a geographic resiliency of resources in theplurality of allocation solutions; based on the resiliency score and theresiliency requirement, allocating, by the processing system, resourcesto the instantiation, wherein the resources comprise a first resource ofthe global tier and a second resource of the regional tier; responsiveto the allocating the resources to the instantiation, periodically:updating, by the processing system, the resource map of the cloudnetwork to generate a predictive heat map of free resources of the cloudnetwork; and determining, by the processing system, a plurality ofpotential reallocation solutions for mitigating potential resourcefailures affecting the network function according to the predictive heatmap of free resources of the cloud network; detecting, by the processingsystem, a first resource failure affecting the network function; andimplementing, by the processing system, a first reallocation solution ofthe plurality of potential reallocation solutions responsive to thedetecting the first resource failure affecting the network function. 9.The method of claim 8, wherein the resource requirements furthercomprise at least one of an affinity rule or a non-affinity rule. 10.The method of claim 8, further comprising: determining, by theprocessing system, a failure or unavailability of at least a subset ofthe resources; and reallocating the instantiation based on the at leastone of an affinity rule or a non-affinity rule.
 11. The method of claim8, wherein the resource map further comprises an edge tier, wherein alatency of the edge tier is lower than a latency of the global tier. 12.The method of claim 8, wherein the global tier comprises at least oneglobal datacenter and the regional tier comprises at least one regionaldatacenter.
 13. The method of claim 8, further comprising: comparing, bythe processing system, the resource requirements to a second resourceneed of a second network function; and identifying, by the processingsystem, a first allocation solution and a second allocation solutionbased on the resource requirements, the second resource need, and theresource map, wherein the allocation solution is based on selecting thefirst allocation solution over thesecond allocation solution.
 14. Themethod of claim 13, wherein the first allocation solution has a firstreliability score that is higher than a second reliability score of thesecond allocation solution.
 15. The method of claim 13, wherein thefirst allocation solution satisfies a geographic preference associatedwith the instantiation.
 16. A non-transitory computer readable storagemedium storing instructions that cause a processor executing theinstructions to effectuate operations, the operations comprising:receiving a request to allocate an instantiation of a network function,wherein the request comprises information indicative of resourcerequirements of the instantiation, wherein the resource requirementscomprise at least one resiliency requirement, and wherein the at leastone resiliency requirement comprises selecting one or more datacentersthat provide a similar amount of control in a region and each of the oneor more datacenters being associated with a similar latency to managededge network components in the region; computing a resource map of acloud network comprising hierarchy of datacenters; comparing theresource requirements with the resource map to determine a plurality ofallocationsolutions; computing a resiliency score for each of theplurality of allocation solutions; based on the resiliency score and theresiliency requirement, allocating resources of thedatacenters to theinstantiation, wherein the resources satisfy the at least one resiliencyrequirement; responsive to the allocating the resources to theinstantiation, periodically: updating the resource map of the cloudnetwork to generate a predictive heat map of free resources of the cloudnetwork; and determining a plurality of potential reallocation solutionsfor mitigating potential resource failures affecting the networkfunction according to the predictive heat map of free resources of thecloud network; detecting a first resource failure affecting the networkfunction; and implementing a first reallocation solution of theplurality of potential reallocation solutions responsive to thedetecting the first resource failure affecting the network function. 17.The non-transitory computer readable storage medium of claim 16, whereinthe hierarchy comprises a global tier and a regional tier.
 18. Thenon-transitory computer readable storage medium of claim 16, wherein theat least one resiliency requirement comprises at least one of a localresiliency and a geographic resiliency.
 19. The non-transitory computerreadable storage medium of claim 16, wherein the operations furthercomprise: allocating second resources of the datacenters to a secondinstantiation of a second network function.
 20. The non-transitorycomputer readable storage medium of claim19, wherein the operationsfurther comprise: detecting a failure affecting at least one of theresources and at least one of the second resources; and determining thatthe second instantiation has a higher priority than the instantiation;andreallocating third resources of the datacenters to the secondinstantiation prior to reallocating fourth resources of the datacentersto the instantiation.